Skip to content

Add conformant LZC compression and NuFX LZC threads - #131

Open
Hawkynt wants to merge 1 commit into
mainfrom
fix/lzc-block-packing
Open

Add conformant LZC compression and NuFX LZC threads#131
Hawkynt wants to merge 1 commit into
mainfrom
fix/lzc-block-packing

Conversation

@Hawkynt

@Hawkynt Hawkynt commented Aug 31, 2026

Copy link
Copy Markdown
Owner

What changed

  • add a clean-room managed LzcCodec for native UNIX compress / .Z streams;
  • implement the format's required LSB-first eight-code group packing, including group flush/alignment at code-width changes and block CLEAR boundaries;
  • support 9-16 bit streams, block/non-block decoding, strict header validation, malformed future-code rejection, truncation/expanded-length checks, and CLEAR-code decoding;
  • add LzcBuildingBlock to the existing source-generated building-block registry surface;
  • route FileFormat.Compress through the conformant LZC primitive instead of the generic continuous-bit LZW helper;
  • add NuFX formats 4/5 as real LZC-12/LZC-16 read/write methods, including native 1F 9D .Z framing inside the NuFX thread;
  • preserve LZC-12/LZC-16 during direct NuFX record replacement and include them in explicit/auto method selection and integrity validation.

Design decisions

The existing generic LzwEncoder/LzwDecoder remains unchanged. LZC's group flush/alignment behavior is wire-format semantics, not merely another bit-order option, so forcing it into the continuous-bit helper would make that abstraction misleading.

The encoder emits valid block-mode streams without adaptive ratio-based CLEAR heuristics. Once the dictionary fills it remains fixed at the configured maximum width; the decoder accepts CLEAR codes from adaptive encoders. This keeps the implementation deterministic while remaining interoperable.

Native .Z has no expanded-length field. The benchmarkable IBuildingBlock envelope therefore prefixes a 32-bit little-endian length before the otherwise-native 16-bit block-mode stream; the concrete LzcCodec API operates on true .Z bytes.

NuFX LZC threads deliberately retain the three-byte .Z header. NufxLib writes and reads that framing for LZC; this differs from NuFX Squeeze threads, where the standalone Squeeze header is omitted.

No new package, native library, P/Invoke, unsafe code, or external executable dependency is introduced.

Validation

Added NUnit coverage for:

  • fixed LZC-12 and LZC-16 byte vectors for TOBEORNOTTOBEORTOBEORNOT;
  • a mid-group CLEAR vector proving discarded group padding/alignment;
  • 12/16-bit dictionary growth across code-width transitions and dictionary saturation;
  • non-block-mode width transitions;
  • empty streams;
  • reserved-header flags, future dictionary codes, truncation, and enclosing-format max-width mismatch;
  • CompressStream interoperability/alignment regression coverage;
  • NuFX LZC-12/LZC-16 creation, extraction, native thread framing, integrity validation, and direct-replacement method preservation.

The fixed .Z vectors and generated width-transition/block/non-block streams were checked with GNU gzip -cd as an independent decoder oracle. This environment does not provide the .NET SDK, so repository compilation/test execution is delegated to PR CI rather than claimed locally.

References and licensing

Consulted:

  • CiderPress II LZC notes for .Z header fields, variable-width coding, eight-code group packing, width-change/CLEAR alignment, and NuFX usage;
  • NufxLib Lzc.c / Compress.c for NuFX format identifiers and confirmation that LZC threads carry native .Z framing;
  • GNU gzip as a behavioral interoperability oracle.

NufxLib is BSD-licensed and the legacy compress implementation it incorporates identifies the relevant historical code as public domain, but this change does not copy or closely translate that source. The implementation is independently written from the documented behavior and validated with external byte-level oracles.

@Hawkynt
Hawkynt force-pushed the fix/lzc-block-packing branch from 226c1b3 to d86bef7 Compare August 31, 2026 08:40
UNIX compress does not use a continuous variable-width LZW bitstream: codes are buffered in groups of eight and a width change or CLEAR discards the rest of the old group. Implement that framing once in Core, route the standalone .Z format through it, and enable NuFX LZC-12/LZC-16 threads without silently changing their compression method on replacement.

Add fixed byte vectors, CLEAR/alignment, width-transition, malformed-stream, standalone-format and NuFX round-trip coverage. Refresh the generated Core package API documentation for the new public building block.
@Hawkynt
Hawkynt force-pushed the fix/lzc-block-packing branch from d86bef7 to 3fb2a23 Compare August 31, 2026 16:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant